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ARTICLE INFO ABSTRACT 

This study introduces a system design in a form of cognitive dialog 
game (DiaCog) to support pedagogical factors and student learning model 
ideas. The purpose of the study is to describe how such a design achieves 
tracking and adapting students’ knowledge and mastery learning levels as 
a cognitive assistant. Also, this study shows alternative ways for supporting 
intelligent personal learning, tutoring systems, and MOOCS. This paper 
explains method called DiaCog that uses structure for students’ thinking 
in an online dialog by tracking student's level of learning/knowledge status. 
The methodology of computing is the semantic that match between students’ 
interactions in a dialog. By this way it informs DiaCog’s learner model to 
inform the pedagogical model. Semantic fingerprint matching method 
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1. INTRODUCTION 


about failing to provide a social environment 
that enables having constructive cognitive 


Intelligent cognitive assistants and tools 
can be implemented in e-learning platforms 
and massive open online source courses sys- 
tems (MOOCS) to reduce the educational 
costs and improving life quality of people and 
to address the educational needs of people who 
are unable to leave their home due to their care 
needs. Although such users can easily access 
to current MOOCS platforms, they may not 
fully benefit from them since there are some 
technical challenges to overcome. For exam- 
ple, MOOCS platforms are usually criticized 
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feedback opportunities in a sustained engage- 
ment during the online courses (Nkuyub- 
watsi, 2013; Kop et al., 2011; Zapata, 2010). 

Discussion forums are used where stu- 
dents have most of the social interaction to 
help each other and to discuss ideas related to 
the learning topics in MOOCS (Ezen-Can et 
al., 2015). Discussing ideas on learning topic 
have positive impacts on students learning 
and building knowledge in online platforms 
(Palmer et al., 2008). 

On the other hand, we observe that dis- 
cussion forums bring several issues in learn- 
ing despite their positive benefits on learning. 
For example, a poor designed and implement- 
ed discussion forum activity may not support 
to discuss at all. The engagement of students 
may be very superficial (Bain, 2011; Hawkey, 
2003; Thomas, 2002). Thus, discussion on on- 
line platforms may provide support for learn- 
ing but it is not guaranteed that all the partici- 
pants of the dialog may join to the ongoing 
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discussion or the discussion 1s deep and mean- 
ingful. There is still have more to explore in 
students’ dialog in technology supported ap- 
plications. Especially, we need to know which 
students actively and meaningfully participate 
to the dialogs and which of them learn from 
the dialog as the knowledge building activities 
emerges in ongoing dialogs. 

Using new cognitive assistants can be 
supported with contemporary dialog tech- 
nologies. By this way, it may be possible to 
improve students’ positive learning results 
by tracking meaningful dialog participation 
and interaction. To enable these functions, it 
is possible to design modern applications that 
track and adapt knowledge level of students. It 
may allow us to closely understand students’ 
dialog-based learning interactions. 

To address the need of designing new 
applications as described here, this study dis- 
cusses the design of a modern learning tool 
that is called as DiaCog. The remaining of the 
paper presents how DiaCog design enables al- 
ternative dialogic interactions and modeling 
learning processes in a meaningful way. 


2. MATERIALS AND METHODS 


2.1. Designing Interaction and 
Interface 


Table 1 shows a part of a hypothetical 
dialog. This dialogue is an example of inter- 
action between players in a typical dialogue 
game. Dialog may go in a way that the play- 
ers may reply to previous entries so that there 
is no linear order in the dialog. Therefore, the 
ongoing dialog cannot be read as reading a 
page of a book in a top to down direction be- 
cause the events are non-linear. The rounds are 
indicated by round numbers in the first bracket 
and the dialog moves are coded in the second 
bracket at the end of players’ entries. 


Table 1. An Example Dialog 


START: Copyright laws and policy could be better 
aligned with the interests of both consumers and 


copyright holders. 


Player A: I agree, although the Copyright Act at- 
tempts to balance culture and commerce through ex- 
clusive incentive models and fair use defenses, the law 
just doesn’t seem to be keeping up with the way end 
users, developers, and content creators operate in the 


digital sphere [Round 1], [Agree]. 


Player D: For instance, current copyright laws are 
inadequate for the digital age anyway, some legal 
experts says "Most of it was written more than a 


quarter century ago," [Round 1], [Support]. 


Player A: For example, we see it all the time on 
YouTube: people communicating through shared 


content without permissions. [Round 3], [Agree]. 


Player B: Is it the case that, the Copyright Act pro- 
tects “original expression,” but what is considered 


“original”? [Round 1], | Question]. 


Player C: I read that, a work can only be original if it 
is the result of independent creative effort. It will not 
be original if it has been copied from something that 
already exists. If it is sumilar to something that already 
exists but there has been no copying from the existing 
work either directly or indirectly, then it may be 
original. [Round 1], [Inform]. 


Player B: I am not sure about that, copyright protec- 
tion should not depend on the artistic quality or merit 


of a work [Round 2], [Challenge]. 


Player C: I read that, the term "original" also involves 
a test of substantiality - literary, dramatic, musical and 
artistic works will not be original if there has not been 
sufficient skill and labor expended in their creation. 
But, sometimes significant investment of resources 
without significant intellectual input can still count as 


sufficient skill and labor. [Round 2], [Support]. 


DiaCog has essential functions such as 
a)interface interactions tied to the meaning 
that the users wish to communicate, b)dialog 
roles and specific rooms, c)structured moves 
categorized as interface elements, and d)dia- 
log rules in order to carry the user’s goals (e.g. 
turn taking). These functions create the basic 
environment for successful collaborative ar- 
gumentations that teachers may apply dialog- 
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based learning scenarios to support students in 
practicing their argumentation skills and cre- 
ative thinking skills. 

Figure | illustrates the typical design of 
DiaCog’s dialog game window. In Figure 1, 
participants are required to select moves from 
the list of available speech acts (see Label # 
1) and related openers (see Label#2) and type 
a free text of their response into the text box 
(see Label #3). After finishing building the 
expressions, participants submit their replies 
once they click the “Act” button (“See Label 
#4). 








Select a related 


is h n 
openers ype the response 









@ Select Your Moves * 







I read that 
3 test of su 


» the term “original” also involves 
bstanti - 
| and artisti 





Click “ACT” button to 


Choose moves send reply 


Figure 1. The typical design of Dia- 
Cog’s dialog game 


2.2. Design of Pedagogical and Learner 
Models in DiaCog 


Student state tracking and adaptation is 
a common application in intelligent tutoring 
system (ITS), which can guide us for the design 
decisions for cognitive assistant implementa- 
tions. In the context of modern ITS design, the 
educational needs of students are tracked and 
adapted by domain model, pedagogical mod- 
el and learner model. Domain model (DM) 
tracks the set of skills, atomic components of 
knowledge objects and content-based strate- 
gies. Learner model(LM) tracks cognitive, 
motivational, affective and other psychologi- 
cal states of learners during the tutoring activi- 
ties. Pedagogical model (PM) decides tutoring 
strategies and course of action by taking input 
from LM and DM. 


Modify LM 








Optimize Tutor Actions 


Adaptation of Knowledge - Content 


Figure 2. Presentation and relations of 
LM, PM. adn DM 


As Figure 2 demonstrates, since LM and 
DM feed PM, it is a better to decide the PM 
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design first before deciding for the LM design 
(DM will not be discussed here since it may 
dramatically change due to the topic and learn- 
ing domain which is difficult to comprehend 
in the study). Following section describes the 
pedagogical strategies for designing PM. 


2.3. Pedagogical Model 


To tackle the design questions of select- 
ing a suitable learning model for MOOC, we 
form relationships between domain model 
(DM), pedagogical model (PM) and learning 
model (LM). Then, we need to analyze the 
educational needs of typical students to build 
PM. For PM design, we list learning theories 
and values to track in Table 2 and Table 3 
based on an extended version of Sottilare et 
al (2013). 

Table 2. Followed theories for Pedagog- 
ical Model 


Difficulty 
Proximity 
Self Regulation Progress 
Theory: Effort 
Goal Setting, Effectiveness 
Self- Assessment, Perceived success (strength 
Self Reflection /weakness) 
Personal Interest 
Improvement and Traits 
Social and 
— Attention 
Constructivist 
Interest (General) &Interest 
Theory: 


(Specific) 
Recruitment, De- . 
Goal setting — Goal commitment 
gree of Freedom, - 
Ability for Degree of Freedom 
Task (domain) difficulty 


Direction, Critical 


Features, Frustra- 


On/off task behavior 
tion control, 
Affective States and traits 
Demonstration, 
Session difficulty 


Teachable — peer 
Patterns of interaction 


agents. 
Style of language 
Motivation (Self ea e 
. Type of feedback (reaction) 
Determination) 
Positive Trust 
Theory 


Satisfaction 

Free decision making 
Autonomy, Rel- 

Preferences and interest 

Value 

(Gagné & Deci, 2005) 


Attention 


evance, Compe- 


tence 


Interest (Situa- 
Levels of learning 


tional) Theo 
= Goals 
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Table 3. Followed values and variables 


for Pedagogical Model 
7 Item Rating 
Cognitive 
| Trails 
load and 


Error/Misconceptions 
item 
success rates 


difficulty | 
Time spent (latency) 
Intellectual — 
| Intellectual abilities /prior knowledge 
Traits 
Accommodator 
Converger 
Diverger 
Assimilator 
(Kolb, 1985) 
Visual learning 
Auditory learning 
Learning ; 
Read/write learning 
styles 


Kinesthetic learning 
(Rosenberg & Burkert 2015) 
Active/Reflective 
Sensing/Intiutive 
Visual/Verbal 
Sequential/Global 
(Felder&Soloman,2014) 


As given in Figure 2, considering the 
roles of different models in system design, 
cognitive assistant adaptation and_ track- 
ing related modules may take large num- 
ber of variables to adaptn in theory. How- 
ever, in practice, the numbers of variables 
are relatively small due to scale up issues. 

PM can be designed by using different 
pedagogical strategies as presented in Table 
3. In our design, we selected a purposeful 
PM strategy that is fitting the nature of the 
pedagogical needs of students’ that are learn- 
ing by engaging interactive dialogs in dis- 
cussions. For this purpose, specifically con- 
sidering the dialog-based model of Di-aCog, 
selected PM should track self-regulation and 
social constructivist educational needs (and 
linked values) since these are suitable for 
learning by dialogs strategies in where the 
learner arrives at solutions through active 
participation in dialog interactions. In Dia- 
Cog, tracking and adaptation of these values 
are achieved by LM, which feed the PM. 
The design process of LM and methods that 
LM uses explained in the following section. 


2.4. Learner Model 


Learner models usually infer the diag- 
nosis like a feature to represent the current 
knowledge state of the students including 
learning difficulties and misconceptions (Co- 
nati, 2002). The learner model is dynamically 
updated as students build knowledge thus the 
model represents about latest understanding 
of the students (Bull, 2004). Learner mod- 
els describes the (1) cognitive processes that 
governs students’ interactions, (2) knowledge 
gap levels between the student and expert, (3) 
students’ behavior patterns, (4) students’ char- 
acteristics and profiles (Webb, et al., 2001). 
Using predictive learner models, it is possible 
to have an insight into the nature of the dia- 
logical interactions and learning. Using these 
models, each reasoning step can be traced, and 
the misconceptions can be detected (Johnson 
and Taatgen, 2005). 

DiaCog uses a dialog-based learning 
strategy as described and modeled in PM. 
Thus, the learner model also should be dia- 
log based to match with PM. In dialog based 
learning models, analyzing the knowledge 
level and learning status in ongoing dialog of 
students could model students learning. Basic 
natural language dialogue analysis techniques 
such as calculating student turns, number of 
words used in dialog turns, percentage of 
words per turns by students are indication of 
learning in an online platform (Core, 2003; 
Rose et al., 2003; Huang et al., 2015). More 
sophisticated techniques such as analyzing the 
different types of speech acts or dialog moves, 
structure of discourse and contextual mean- 
ing are also possible for tracking the learn- 
ing in dialog (Sardareh et al., 2014; Prylipko 
et al., 2014; Vail, 2014; Rotaru, et al., 2006). 

The power of learner model may change 
due to the sophistication of linguistic tech- 
niques that are applicable for tracking modules 
in MOOCS (Forbes-Riley, 2007). For instance, 
some techniques are easy to apply but not 
meaningfully informative (e.g. turn counts, di- 
alog length) while others provide much worth- 
while data for modeling (e.g. speech acts) 
(Bernsen, et al., 2012). Some of the techniques 
are well informative and largely available in 
the ITS literature to apply in for MOOCS but 
they require more effort like labeling dialog 
acts or designing computational algorithms 
for dialog analysis and labeling of corpus. 

In DiaCog, we aim to tackle the issue of 
understanding deeper pragmatic properties of 
dialog by providing combination of several so- 
phisticated methodologies that enable analyz- 
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ing the dialog yielding stronger learner mod- 
els. Using combining techniques in DiaCog’s 
proposed LM design it is possible to have so- 
phisticated tracking functions in MOOCS sys- 
tems. In DiaCog, LM may be improved further 
for additional applications for more advanced 
automatic adaptive MOOC functions that can 
provide more specific personalized and _ tai- 
lored instruction in addition to the pedagogical 
strategies that PM delivers as discussed above. 

LM strategies for DiaCog can be con- 
structed using machine-learning approaches 
(Webb, et al., 2001). However, requirement 
of explicitly labeled data is one of the biggest 
challenges of machine learning approaches 
for learner modeling and may be challenging 
for DiaCog’s LM design. Because the cor- 
rect labels may not be available from simple 
observation of dialogs, these labels may re- 
quire a manual coding or an algorithm for 
label clustering (Fereday et al., 2006). After 
determining correct labels, speech acts may 
represent deeper pragmatic properties in dia- 
log, which informs the PM about students’ 
status. Fortunately, DiaCog interface pro- 
vides a partially shortcut solution for the is- 
sue of manual labeling of speech acts. Since 
DiaCog interface requires users to indicate 
their intention using speech acts specially 
designed and labelled as “dialog moves” in 
the dialog interaction interface, DiaCog eas- 
ily understand and categorize the speech acts 
when users build their expressions in ongo- 
ing dialogs (Yengin and Lazarevic, 2014). 

Tracking students’ learning (knowledge 
status) and comparing with expert knowl- 
edge universe handled in DiaCog as illus- 
trated in Figure 3. To track students’ learn- 
ing and knowledge status, DiaCog compute 
the semantic match between all ideal student 
interactions in a dialog and compares it with 
an expert knowledge domain map encoded. 
Comparing students’ current state of learn- 
ing and knowledge of content with expert 
knowledge rule spaces is mapped in a QMA- 
TRIX, which enables learning-knowledge 
level for tracking students (Lee and Sawak1, 
2009). The use of QMATRIX is relatively 
well-known technique in rule space learner 
models such as the “Additive Factor Model” 
(AFM) (Li et al., 2011) that is using the “Item 
Response Theory” (Embretson and Reise, 
2013). AFM is good model for detection of 
students’ prior knowledge, which can be used 
to predict later performances. DiaCog’s LM 
design uses same principles for this purpose. 
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Student Knowledge 





Figure 3. LM Design for DiaCog 


To map student knowledge and com- 
pare it with expert knowledge in QMATRIX, 
DiaCog needs to know whether students have 
similar knowledge structures with the expert 
domain knowledge. In simple words, DiaCog 
needs to compare the student dialogs with the 
expert knowledge. If there is a match between 
student knowledge and expert knowledge; it 
is mapped in QMATRIX using variables that 
show the student knows a knowledge or con- 
tent object. If there is no match, this is mapped, 
as the student has no knowledge. If there 1s a 
match leading a possible misconception, this 
is mapped as misconception in QMATRIX. 

To understand if there is any matching 
or not, DiaCog must analyze the student in- 
put texts in ongoing dialogs. This function is 
carried in DiaCog by a “semantic matching” 
engine that measures semantic relatedness us- 
ing a direct measure which is a stronger and 
sophisticated technique rather than words 
analysis. DiaCog “semantic matching” engine 
uses “cortical.1o Retina API’ services to make 
direct semantic comparison of the meanings 
stored in students’ dialogs. Using semantic re- 
lations, DiaCog is capable to measure whether 
participants of dialogs talk (know) about on 
the same concepts with the expert domain 
knowledge universe by applying natural- 
language processing techniques based on the 
distributed representations of text segments 
grounded in a neuro-computational model of 
semantics (Corticalio, 2015; Chi, 2009). This 
allows DiaCog to interpret students’ conceptu- 
al understanding and their meaning closeness. 
Figure 4 shows a conceptual dialog mapping 
of students. 
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QMATRIX 
Expert Knowledge Universe Space 


Knowledge of Apple (Computer) 


Knowledge of 
Apple (Fruit) ~ 


Apple - Fruit = Computer 


Retina API Calculates 
Expression 

(Used for misconceptions 
calculations in QMATRIX) 


Misconceptions 





Figure 4. A conceptual dialog mapping 
of students 


According to the example QMATRIX in 
Figure 4, student knowledge of apple belongs 
to the knowledge space of fruits. On the other 
hand, the expert knowledge domain requires 
student to know/learn the apple as a concept of 
computer brand Apple (Gust for giving an ex- 
ample purpose). To detect and understand this 
student’s knowledge level (concept of apple) 
and the potential misconception, DiaCog’s LM 
should match the expert knowledge space with 
students’ knowledge level. For this purpose, 
semantic matching engine calculates the dis- 
tance scores for these knowledge objects (ap- 
ple as fruit and Apple as computer brand) and 
compare the scores to see how close they are 
to each other. If the closeness score 1s less, this 
means student is very close to expert knowl- 
edge space and well knows the knowledge ob- 
ject. If there is a great distance than that means 
student doesn’t know the knowledge object. If 
there is an overlap like fruit domain and com- 
puter, semantic engine calculates the overlap. 
The degree of distance inexpert and student 
semantic relatedness scores are calculated us- 
ing to the median score of overall population 
score in the system. Finally, if the overlap is 
not close to expert knowledge space, then it 1s 
labeled as misconception and encoded 1n this 
space. 


3. RESULTS 


In this study, design of DiaCog applica- 
tion aimed for implementation as cognitive 
assistant in MOOCS and discussed DiaCog’s 
functions for enabling tracking students’ 
knowledge levels and behavior in dialogs. 
The suggested pedagogical model in DiaCog 
explains strategies fitting the nature of learn- 
ing with dialogs in discussions. Matching the 


pedagogical model, DiaCog’ learner model is 
demonstrated how to use the semantic match 
technique to compare student knowledge and 
learning with an expert knowledge domain 
map to enable tracking learners’ knowledge 
when they engage discussion in cognitive as- 
sistant supported MOOCS. These design de- 
cisions and techniques for pedagogical and 
learning models let DiaCog to be an alterna- 
tive module as cognitive assistants for track- 
ing expert knowledge and detect student mas- 
tery levels in learning. 


4. DISCUSSIONS 


DiaCog’s special design and analytical 
functions opens a door for learner modeling 
enabling knowledge tracking in dialogs. In fu- 
ture, we plan to train a student and learning 
model using linguistic features derived from 
DiaCog by applying machine-learning algo- 
rithms to create a model of students thinking 
processes in dialogs that also can be applied 
to simulate typical thinking patterns in future 
dialogs then. Using such models as cognitive 
assistant implementations in MOCCS may 
have adaptive pedagogical strategies to the 
students’ state to improve learning experience 
by providing additional feedback or course 
materials dynamically to the situations. Also 
predictive models based on machine learning 
algorithms may enable MOOCS to understand 
the users’ knowledge level positions in differ- 
ent learning scenarios. In addition, cognitive 
assistants enabled MOOCS can determine fu- 
ture skill levels of students so it can have a 
control and guiding mechanism for the course 
of learning through interactions in dialog or 
other content. 

In future, we are also planning to exam- 
ine how well different student models perform 
and generalize with different user populations 
using MOOCS. We will test the effectiveness 
of the DiaCog on learning by conducting ex- 
periments to validate the success of implemen- 
tation and pedagogic usefulness of DiaCog in 
real life. 


5. CONCLUSIONS 


In conclusion, this paper showed and 
explained the design of cognitive dialog game 
(DiaCog) as cognitive assistants. The tracking 
and adapting abilities of DiaCog as a cogni- 
tive assistant was illustrated. The methodol- 
ogy of computing the semantic that match be- 
tween student interactions in a dialog is used 
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to inform DiaCog’s learner model. Semantic 
fingerprint matching method used to make 
comparisons with expert knowledge in order 
to detect student mastery levels in learning. 

As a result, DiaCog tool and applied 
methodologies can be used for implementing 
pedagogical and learner modeling to track and 
adept students’ learning. 

Finally, this paper discussed the im- 
provements and possible experimental designs 
to advance the techniques. 
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